inform theori branch appli mathemat electr engin involv quantif inform inform theori develop claud shannon find fundament limit signal process oper compress data reliabl store commun data incept broaden find applic area includ statist infer natur languag process cryptographi gener network commun network neurobiolog evolut function molecular code model select ecolog thermal physic quantum comput plagiar detect form data analysi kei measur inform entropi express averag number bit need storag commun entropi quantifi uncertainti involv predict random variabl outcom fair coin flip equal outcom inform lower entropi outcom roll die equal outcom applic fundament topic inform theori includ lossless data compress zip file lossi data compress mp channel code dsl line field intersect mathemat statist comput scienc physic neurobiolog electr engin impact crucial success voyag mission deep space invent compact disc feasibl mobil phone develop internet studi linguist human percept understand black hole numer field citat need import field inform theori sourc code channel code algorithm complex theori algorithm inform theori inform theoret secur measur inform content hide overview histor background quantiti inform entropi joint entropi condit entropi equivoc mutual inform transinform kullback leibler diverg inform gain quantiti code theori sourc theori rate channel capac capac channel model applic field intellig secreci applic pseudorandom number gener seismic explor miscellan applic applic histori theori concept refer footnot classic work journal articl textbook inform theori book extern link refer edit overviewth main concept inform theori grasp widespread mean human commun languag import aspect concis languag common word shorter common word benefit gener mediocr sentenc long tradeoff word length analog data compress essenti aspect sourc code part sentenc unheard misheard due nois pass car listen glean mean underli messag robust essenti electron commun system languag properli build robust commun channel code sourc code channel code fundament concern inform theori note concern import messag platitud take long write urgent plea call ambul import meaning context inform theori messag import mean matter qualiti data quantiti readabl data determin sole probabl inform theori gener consid found claud shannon semin work mathemat theori commun central paradigm classic inform theori engin problem transmiss inform noisi channel fundament result theori shannon sourc code theorem establish averag number bit need repres result uncertain event entropi shannon noisi channel code theorem state reliabl commun noisi channel provid rate commun threshold call channel capac channel capac approach practic encod decod system inform theori close collect pure appli disciplin investig reduc engin practic varieti rubric world past half centuri adapt system anticipatori system artifici intellig complex system complex scienc cybernet informat machin learn system scienc descript inform theori broad deep mathemat theori equal broad deep applic vital field code theori code theori concern find explicit method call code increas effici reduc net error rate data commun noisi channel limit shannon prove maximum channel code roughli subdivid data compress sourc code error correct channel code techniqu case year find method shannon work prove class inform theori code cryptograph algorithm code cipher concept method result code theori inform theori wide cryptographi cryptanalysi articl ban inform histor applic inform theori inform retriev intellig gather gambl statist music composit edit histor backgroundmain articl histori inform theori landmark event establish disciplin inform theori brought worldwid attent public claud shannon classic paper mathemat theori commun bell system technic journal juli octob prior paper limit inform theoret idea develop bell lab implicitli assum event equal probabl harri nyquist paper factor affect telegraph speed theoret section quantifi intellig line speed transmit commun system give relat klogm speed transmiss intellig number voltag level choos time step constant ralph hartlei paper transmiss inform word inform measur quantiti reflect receiv abil distinguish sequenc symbol quantifi inform logsn nlog number symbol number symbol transmiss natur unit inform decim digit renam hartlei honour unit scale measur inform alan ture similar idea part statist analysi break german world war enigma cipher mathemat inform theori event probabl develop field thermodynam ludwig boltzmann willard gibb connect inform theoret entropi thermodynam entropi includ import contribut rolf landauer explor entropi thermodynam inform theori shannon revolutionari groundbreak paper work substanti complet bell lab end shannon time introduc qualit quantit model commun statist process underli inform theori open assert fundament problem commun reproduc point approxim messag select point idea inform entropi redund sourc relev sourc code theorem mutual inform channel capac noisi channel includ promis perfect loss free commun noisi channel code theorem practic result shannon hartlei law channel capac gaussian channel bit fundament unit inform edit quantiti informationmain articl quantiti inform inform theori base probabl theori statist import quantiti inform entropi inform random variabl mutual inform amount inform common random variabl quantiti easili messag data compress find commun rate channel choic logarithm base formula determin unit inform entropi common unit inform bit base binari logarithm unit includ nat base natur logarithm hartlei base common logarithm express form consid convent equal justifi logarithm base edit entropi entropi bernoulli trial function success probabl call binari entropi function hb entropi maxim bit trial outcom equal probabl unbias coin toss entropi discret random variabl measur amount uncertainti suppos transmit bit bit ahead transmiss absolut probabl logic dictat inform transmit equal independ bit inform theoret sens transmit extrem inform quantifi set messag xn probabl entropi defin inform entropi contribut individu messag expect import properti entropi maxim messag messag space equiprob unpredict case logn special case inform entropi random variabl outcom binari entropi function logarithm base edit joint entropyth joint entropi discret random variabl entropi pair impli independ joint entropi sum individu entropi repres posit chess piec row column joint entropi row piec column piec entropi posit piec similar notat joint entropi confus cross entropi edit condit entropi equivoc condit entropi condit uncertainti random variabl call equivoc averag condit entropi entropi condit random variabl random variabl care confus definit condit entropi common basic properti form condit entropi edit mutual inform transinform mutual inform measur amount inform obtain random variabl observ import commun maxim amount inform share receiv signal mutual inform rel si specif mutual inform pointwis mutual inform basic properti mutual inform know save averag bit encod compar know mutual inform symmetr mutual inform express averag kullback leibler diverg inform gain posterior probabl distribut prior distribut word measur averag probabl distribut chang recalcul diverg product margin distribut actual joint distribut mutual inform close relat log likelihood ratio test context conting tabl multinomi distribut pearson test mutual inform consid statist assess independ pair variabl asymptot distribut edit kullback leibler diverg inform gain kullback leibler diverg inform diverg inform gain rel entropi compar distribut true probabl distribut arbitrari probabl distribut compress data manner assum distribut underli data realiti correct distribut kullback leibler diverg number averag addit bit datum compress defin distanc metric true metric symmetr satisfi triangl inequ make semi quasimetr edit quantitiesoth import inform theoret quantiti includ nyi entropi gener entropi differenti entropi gener quantiti inform continu distribut condit mutual inform edit code theorymain articl code theori pictur show scratch readabl surfac cd music data cd code error correct code read minor scratch error detect correct code theori import direct applic inform theori subdivid sourc code theori channel code theori statist descript data inform theori quantifi number bit need describ data inform entropi sourc data compress sourc code formul compress problem lossless data compress data reconstruct lossi data compress alloc bit need reconstruct data fidel level measur distort function subset inform theori call rate distort theori error correct code channel code data compress remov redund error correct code add kind redund error correct need transmit data effici faithfulli noisi channel divis code theori compress transmiss justifi inform transmiss theorem sourc channel separ theorem justifi bit univers currenc inform context theorem hold situat transmit user wish commun receiv user scenario transmitt multipl access channel receiv broadcast channel intermediari helper relai channel gener network compress transmiss longer optim network inform theori refer multi agent commun model edit sourc theoryani process gener success messag consid sourc inform memoryless sourc messag independ ident distribut random variabl properti ergod stationar impos gener constraint sourc stochast term studi inform theori edit rateinform rate averag entropi symbol memoryless sourc entropi symbol case stationari stochast process condit entropi symbol previou symbol gener gener case process necessarili stationari averag rate limit joint entropi symbol stationari sourc express give result common inform theori speak rate entropi languag sourc inform english prose rate sourc inform relat redund compress subject sourc code edit channel capacitymain articl channel capac commun channel ethernet cabl primari motiv inform theori telephon mobil landlin channel fail produc exact reconstruct signal nois period silenc form signal corrupt degrad qualiti inform hope commun noisi imperfect channel commun process discret channel simpl model process shown repres space messag transmit space messag receiv unit time channel condit probabl distribut function inher fix properti commun channel repres natur nois channel joint distribut complet determin channel choic margin distribut messag choos send channel constraint maxim rate inform signal commun channel measur mutual inform maximum mutual inform call channel capac capac properti relat commun inform rate bit symbol inform rate code error larg exist code length rate decod algorithm maxim probabl block error transmit arbitrarili small block error addit rate imposs transmit arbitrarili small block error channel code concern find optim code transmit data noisi channel small code error rate channel capac edit capac channel model suggest articl section merg channel capac discuss continu time analog commun channel subject gaussian nois shannon hartlei theorem binari symmetr channel bsc crossov probabl binari input binari output channel flip input bit probabl bsc capac hb bit channel hb binari entropi function base logarithm binari erasur channel bec erasur probabl binari input ternari output channel channel output symbol call erasur erasur repres complet loss inform input bit capac bec bit channel edit applic field edit intellig secreci applicationsinform theoret concept appli cryptographi cryptanalysi ture inform unit ban ultra project break german enigma machin code hasten end wwii europ shannon defin import concept call unic distanc base redund plaintext attempt give minimum amount ciphertext ensur uniqu decipher inform theori lead difficult secret brute forc attack break system base asymmetr kei algorithm commonli method symmetr kei algorithm call secret kei algorithm block cipher secur method assumpt attack break practic amount time inform theoret secur refer method time pad vulner brute forc attack case posit condit mutual inform plaintext ciphertext condit kei ensur proper transmiss uncondit mutual inform plaintext ciphertext remain result absolut secur commun word eavesdropp improv guess plaintext gain knowledg ciphertext kei cryptograph system care correctli appli inform theoret secur method venona project crack time pad soviet union due improp reus kei materi edit pseudorandom number generationpseudorandom number gener wide comput languag librari applic program univers unsuit cryptograph evad determinist natur modern comput equip softwar class improv random number gener term cryptograph secur pseudorandom number gener requir extern softwar random seed work intend obtain extractor carefulli measur suffici random extractor min entropi relat shannon entropi nyi entropi nyi entropi evalu random cryptograph system relat distinct measur random variabl high shannon entropi necessarili satisfactori extractor cryptographi edit seismic explorationon earli commerci applic inform theori field seismic oil explor work field made strip separ unwant nois desir seismic signal inform theori digit signal process offer major improv resolut imag clariti previou analog method edit miscellan applicationsinform theori applic gambl invest black hole bioinformat music edit mathemat portal commun theori list import public philosophi inform edit applicationscryptanalysi cryptographi cybernet entropi thermodynam inform theori gambl intellig inform gather seismic explor edit historyhartlei histori inform theori shannon timelin inform theori yockei edit theorycod theori detect theori estim theori fisher inform inform algebra inform asymmetri inform geometri inform theori measur theori kolmogorov complex logic inform network code philosophi inform quantum inform scienc semiot inform theori sourc code edit conceptsban inform channel capac channel commun commun sourc condit entropi covert channel decod differenti entropi encod inform entropi joint entropi kullback leibler diverg mutual inform pointwis mutual inform pmi receiv inform theori redund nyi entropi inform unic distanc varieti 